Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

Disable ENABLE_TESTCOVERAGE on CentOS 7 build #19507

Merged
merged 2 commits into from
Nov 10, 2020
Merged

Disable ENABLE_TESTCOVERAGE on CentOS 7 build #19507

merged 2 commits into from
Nov 10, 2020

Conversation

leezu
Copy link
Contributor

@leezu leezu commented Nov 9, 2020 •
edited
Loading

Causes significant slowdown in connection with the system (devtoolset) provided OpenMP libraries.
#17559 starts pulling in the system OpenMP in addition to the 3rdparty/openmp causing the slowdown to materialize after the merge of #17559.

Fixes #19502

Let's still fix the issue of intgemm pulling a second openmp @kpuatamazon

With this PR

[2020-11-10T00:14:44.881Z] ============================= slowest 50 durations =============================

[2020-11-10T00:14:44.881Z] 467.92s call     tests/python/unittest/test_random.py::test_random

[2020-11-10T00:14:44.881Z] 74.58s call     tests/python/unittest/test_operator.py::test_image_normalize

[2020-11-10T00:14:44.881Z] 74.46s call     tests/python/unittest/test_random.py::test_randint_generator

[2020-11-10T00:14:44.881Z] 53.77s call     tests/python/unittest/test_random.py::test_shuffle

[2020-11-10T00:14:44.881Z] 29.70s call     tests/python/unittest/test_gluon.py::test_slice_pooling2d_slice_pooling2d

[2020-11-10T00:14:44.881Z] 19.73s call     tests/python/unittest/test_random.py::test_dirichlet

[2020-11-10T00:14:44.881Z] 15.36s call     tests/python/unittest/test_sparse_operator.py::test_sparse_mathematical_core

[2020-11-10T00:14:44.881Z] 10.90s call     tests/python/unittest/test_gluon.py::test_slice_pooling2d

[2020-11-10T00:14:44.881Z] 10.63s call     tests/python/unittest/test_operator.py::test_big_transpose

[2020-11-10T00:14:44.881Z] 10.30s call     tests/python/unittest/test_sparse_operator.py::test_sparse_square_sum

Without this PR

[2020-11-09T22:46:33.281Z] ============================= slowest 50 durations =============================

[2020-11-09T22:46:33.281Z] 599.70s call     tests/python/unittest/test_gluon.py::test_slice_pooling2d_slice_pooling2d

[2020-11-09T22:46:33.281Z] 246.20s call     tests/python/unittest/test_random.py::test_randint_generator

[2020-11-09T22:46:33.281Z] 219.33s call     tests/python/unittest/test_gluon.py::test_slice_pooling2d

[2020-11-09T22:46:33.281Z] 182.46s call     tests/python/unittest/test_random.py::test_random

[2020-11-09T22:46:33.281Z] 143.63s call     tests/python/unittest/test_sparse_operator.py::test_cast_storage_ex

[2020-11-09T22:46:33.281Z] 125.78s call     tests/python/unittest/test_sparse_operator.py::test_sparse_square_sum

[2020-11-09T22:46:33.281Z] 88.98s call     tests/python/unittest/test_random.py::test_negative_binomial_generator

[2020-11-09T22:46:33.281Z] 69.17s call     tests/python/unittest/test_random.py::test_poisson_generator

[2020-11-09T22:46:33.281Z] 63.23s call     tests/python/unittest/test_gluon.py::test_slice_batchnorm

[2020-11-09T22:46:33.281Z] 62.36s call     tests/python/unittest/test_gluon.py::test_slice_batchnorm_reshape_batchnorm

@leezu
Disable ENABLE_TESTCOVERAGE on CentOS 7 build
7410497 
Causes significant slowdown in connection with the system (devtoolset) provided
OpenMP libraries.
@mxnet-bot
Copy link

Hey @leezu , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [sanity, windows-cpu, edge, centos-gpu, unix-cpu, unix-gpu, centos-cpu, windows-gpu, clang, website, miscellaneous]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@lanking520 lanking520 added the pr-awaiting-testing PR is reviewed and waiting CI build and test label Nov 9, 2020
@leezu leezu requested review from mseth10 and szha and removed request for aaronmarkham and marcoabreu November 9, 2020 22:29
mseth10
mseth10 approved these changes Nov 10, 2020
View reviewed changes
Copy link
Contributor

@mseth10 mseth10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified that centos-cpu duration reduces to ~50mins from ~100mins after this commit. Thanks for the fix @leezu

@leezu
Copy link
Contributor Author

leezu commented Nov 10, 2020

@mseth10 any idea about the difference in tests/python/unittest/test_random.py::test_random?

@mseth10
Copy link
Contributor

mseth10 commented Nov 10, 2020

Seems to be an outlier, all other tests run faster. Should we re-trigger centos-cpu for a second datapoint?

@leezu
Copy link
Contributor Author

leezu commented Nov 10, 2020

I think we can merge the PR and observe the performance on the master branch.

@lanking520 lanking520 added pr-work-in-progress PR is still work in progress pr-awaiting-merge Review and CI is complete. Ready to Merge and removed pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress labels Nov 10, 2020
@leezu leezu merged commit d46dc96 into apache:master Nov 10, 2020
@leezu leezu deleted the 2020-11/centos7-ci branch November 10, 2020 15:38
vidyaravipati pushed a commit to vidyaravipati/incubator-mxnet that referenced this pull request Nov 11, 2020
@leezu
Disable ENABLE_TESTCOVERAGE on CentOS 7 build (apache#19507)
57b7287 
Causes significant slowdown in connection with the system (devtoolset) provided
OpenMP libraries. intgemm USE_OPENMP appears to pull in system openmp
apache@1393602
@access2rohit access2rohit mentioned this pull request Feb 17, 2021
Closed
13 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@mseth10 mseth10 mseth10 approved these changes

@szha szha Awaiting requested review from szha

Assignees
No one assigned
Labels
pr-awaiting-merge Review and CI is complete. Ready to Merge
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

CI slowdown on CentOS CPU related to OpenMP and intgemm
4 participants
@leezu @mxnet-bot @mseth10 @lanking520